Tomáš Pospíšek's Notizblock

Shells that grok JSON

I noticed that a big part (most?) of the sysadmin work consists of parsing data, mostly config files.

Parsing data in particular config files at this time and age means wasting your time! That is because a lot of modern software is using JSON, YAML or TOML or another standard data representation format for configuration (and/or data interchange).

Both YAML and TOML config files can usually be transformed into JSON without config data loss. If a tool wants to use JSON formatted data it just needs to use one of the available JSON parsing libraries that will transform the JSON data into a struct in the tool's native implementation language. And that's it, no parsing necessary!

So a lot of data and config is coming in the form of JSON et al. but the traditional shells, in particular the standard Linux shell bash, do not have a data type to represent arbitrary structs. That is why we are not able to manipulate modern config files in bash with a snap but have to parse them: poorly, with a lot of effort and the resulting "parser" usually being extremely brittle.

This shows that we Unix sysadmin got stuck in the past, where most config files were in random invented ad hoc formats needing custom parsing. That is not the norm today. But our tools are still not able to understand standadized data formats.

We need to move on.

We know that shells are the tool that gets most integration and automation jobs done the quickest. We know about the usefulnes and power of the "universal" JSON format and the readability of YAML, TOML etc. We do need to combine these two abstractions and move into the future. We need shells that are able to handle arbitrary structures as expressed by JSON.

Let's move on, let's move on to shells that understand structured data:

Tomáš Pospíšek, 2020-04-14

Articles